{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 活动热度数据(event_attendees.csv)处理\n",
    "(只取训练集和测试集中出现的用户 ID)\n",
    "\n",
    "数据来源于 Kaggle 竞赛: Event Recommendation Engine Challenge, 根据\n",
    "①events they’ve responded to in the past\n",
    "②user demographic information\n",
    "③what events they’ve seen and clicked on in our app\n",
    "预测用户对某个活动是否感兴趣\n",
    "\n",
    "竞赛官网:\n",
    "https://www.kaggle.com/c/event-recommendation-engine-challenge/data\n",
    "\n",
    "\n",
    "event_attendees.csv 文件: 共 5 维特征  \n",
    "event_id: 活动 ID  \n",
    "yes, maybe, invited, no: 以空格隔开的用户列表, \n",
    "分别表示该活动参加的用户、可能参加的用户, 被邀请的用户和不参加的用户."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import scipy.sparse as ss\n",
    "import scipy.io as sio\n",
    "\n",
    "import pickle\n",
    "\n",
    "from sklearn.preprocessing import normalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总的用户数目超过训练集和测试集中的用户, \n",
    "为节省处理时间和内存, 先去处理 train 和 test, 得到竞赛需要用到的事件和用户. \n",
    "然后对在训练集和测试集中出现过的活动和用户建立新的 ID 索引. \n",
    "先运行 user_event.ipynb, \n",
    "得到活动列表文件: PE_userIndex.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取之前统计好的测试集和训练集中出现过的活动"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of events in train & test: 13418\n"
     ]
    }
   ],
   "source": [
    "# 读取训练集和测试集中出现过的活动列表\n",
    "eventIndex = pickle.load(open('PE_eventIndex.pkl', 'rb'))\n",
    "n_events = len(eventIndex)\n",
    "print('Number of events in train & test: %d' %n_events)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## user_friends.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'event,yes,maybe,invited,no\\n'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "\"\"\"\n",
    "  统计某个活动, 参加和不参加的人数, 计算活动热度\n",
    "\"\"\"\n",
    "# 活动活跃度\n",
    "eventPopularity = ss.dok_matrix((n_events, 1))\n",
    "\n",
    "f = open('event_attendees.csv', 'rb')\n",
    "f.readline() # 跳过表头"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for line in f:\n",
    "    cols = line.strip(). split(','.encode(encoding='utf-8'))\n",
    "    eventId = cols[0]\n",
    "    if eventIndex.__contains__(eventId):\n",
    "        i = eventIndex[eventId] # 活动索引\n",
    "        \n",
    "        # yes-no\n",
    "        eventPopularity[i, 0] = \\\n",
    "        len(cols[1].split()) - len(cols[4].split())\n",
    "        \n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "eventPopularity = normalize(eventPopularity, norm='l1', axis=0, copy=False)\n",
    "sio.mmwrite('EA_eventPopularity', eventPopularity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "matrix([[  8.45412836e-05],\n",
       "        [  4.34130916e-05],\n",
       "        [  7.08318862e-05],\n",
       "        ..., \n",
       "        [ -1.11960078e-04],\n",
       "        [  9.36808818e-05],\n",
       "        [ -7.76865849e-05]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eventPopularity.todense()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
